A fuzzy pattern-based filtering algorithm for botnet detection

Please cite this article in press as: K. Wang et doi:10.1016/j.comnet.2011.05.026 Botnet has become a popular technique for deploying Internet crimes. Although signaturebased bot detection techniques are accurate, they could be useless when bot variants are encountered. Therefore, behavior-based detection techniques become attractive due to their ability to detect bot variants and even unknown bots. In this paper, we propose a behavior-based botnet detection system based on fuzzy pattern recognition techniques. We intend to identify bot-relevant domain names and IP addresses by inspecting network traces. If domain names and IP addresses used by botnets can be identified, the information can be further used to prevent protected hosts from becoming one member of a botnet. To work with fuzzy pattern recognition techniques, we design several membership functions based on frequently observed bots’ behavior including: (1) generate failed DNS queries; (2) have similar DNS query intervals; (3) generate failed network connections; and (4) have similar payload sizes for network connections. Membership functions can be easily altered, removed, or added to enhance the capability of the proposed system. In addition, to improve the overall system performance, we develop a traffic reduction algorithm to reduce the amount of network traffic required to be inspected by the proposed system. Performance evaluation results based on real traces show that the proposed system can reduce more than 70% input raw packet traces and achieve a high detection rate (about 95%) and a low false positive rates (0–3.08%). Furthermore, the proposed FPRF algorithm is resource-efficient and can identify inactive botnets to indicate potential vulnerable hosts. 2011 Elsevier B.V. All rights reserved.